Debug the firmware

Introduction

Sometimes (many times actually), our code won't do what we want it to do and we need to take a look at what it's doing. By using a debugger we will be able to see what is going on inside another program while it executes or even crashes. This is fairly straight forward when you code for a modern day computer, since most IDEs have a proper interface integrated for it. However, debugging a chip like the SAMD21 can sometimes be tricky and here is where it's interesting to use a debugging kit.

Image Credit: XKCD

To keep it simple: our final target is to be able to interact with the SAMD21 (or the chip) while it's executing the program and tell it to pause the execution, give us the value of some variables and then continue. We will release a fairly extensive report with documentation on this process, but for those interested in reading an overview on how to debug, this post can be a short introduction.

So, here we go! The first item we need is the Open On-Chip Debugger (OpenOCD) which provides debugging with the assistance of a debug adapter. This adapter is a small hardware module which helps provide the right kind of electrical signaling to the target being debugged. These are required since the debug host, on which OpenOCD runs (i.e. your computer, a Raspberry PI...) won’t usually have native support for such signaling, or the connector needed to hook up to the target.

Image Credit: Smart Citizen

These adapters are sometimes packaged as discrete dongles, which may generically be called hardware interface dongles (and are quite expensive). Some development boards also integrate them directly, which may let the development board connect directly to the debug host over USB (and sometimes also to power it over USB, like the Arduino Genuino Zero). In the case of the Smart Citizen Kit, we have a SWD Adapter that supports Serial Wire Debug signaling to communicate with the ARM core. In our approach, using a complete open toolchain, OpenOCD is be running on a Raspberry Pi, and communicating with the SCK's SWD through the GPIO pins of the Pi.

GDB->OpenOCD: Let's debug the SCK
OpenOCD->GDB: OK!
OpenOCD->SCK: We are debugging you
Note right of SCK: SCK thinks
SCK->OpenOCD: OK!
GDB->OpenOCD: Load firmware
GDB->OpenOCD: Set Breakpoints
OpenOCD->SCK: Do it!
OpenOCD->SCK: Run!
Note right of SCK: SCK runs
Note right of SCK: SCK hits a breakpoint
SCK->GDB: Temperature sensor reading is 500ºC
Note left of GDB: Oh, oh...

Finally, to be able to actually see what is going on inside our firmware while it executes, we need something that is able to read and understand the machine code and hand it over to a human understandable interface. This is where GDB kicks in and helps us by:

• Starting our program, specifying anything that might affect its behavior.
• Make our program stop on specified conditions.
• Examine what has happened when our program has stopped.
• Change things in our program, so we can experiment with correcting the effects of one bug and go on to learn about another.

GDB and OpenOCD will be running in a Raspberry Pi hooked up to the SWD interface of the SCK, and we will see what's going on in them from our computer's terminal via SSH. Fairly simple, right? Now, we can make some changes to our code, make GDB flash it to the SCK and keep debugging in a completely open toolchain!

Debugger setup using a Raspberry Pi

• First download and copy Raspbian Lite to your SDcard, here are the installation docs.

• Add wifi configuration

  Create a file name wpa_supplicant.conf on the /boot partition of the SD card, the content of this file should looks like this:

  ```shell= ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev update_config=1

  network={ ssid="wifi_ssid" psk="wifi_password" } ``` Replacing wifi_ssid and wifi_password with your actual wifi network information. The wpa_supplicant.conf file will be copied to /etc/wpa_supplicant/ directory automatically once the Raspberry Pi is booted up.

• Enable SSH server.

  SSH access is disabled as default for security reasons. To enable the SSH server when Raspberry Pi is booted up for the first time: create a file called ssh with no file extension and no contents, and copy it to the /boot partition on the SD card.

• Find your raspberry on the network

In order to find a raspberry pi over the network we can use commands like these:

Linux ```shell= MY_IP_RANGE=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}') && nmap -sn $MY_IP_RANGE && IP=$(arp -na | grep b8:27:eb | grep -Eo '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}') && ssh $IP;

**Mac** (To be reviewed)
```shell=
MY_RANGE=$(ip addr | grep "UP" -A3 | grep '192' -A0 | awk '{print $2}') && nmap -sn $MY_RANGE && arp -na | grep b8:27:eb

• SSH login without password:

  • Si nunca has generado RSA key: ssh-keygen sin poner nada en la passphrase.
  • Copiar la clave a la raspberry: ssh-copy-id -i ~/.ssh/id_rsa.pub raspi-address

  Ya que esté lista la raspberry con conectarla debe bootear y conectarse sola a la red, el commando que esta arriba (MY_IP...) la localiza y hace ssh login.

Once you are logged to your raspberry pi and connected to the internet, do a system upgrade: ```shell= sudo apt-get install rpi-update sudo rpi-update sudo apt-get update && sudo apt-get dist-upgrade

Install some **dependencies**:

```shell=
sudo apt-get install git autoconf libtool make pkg-config libusb-1.0-0 libusb-1.0-0-dev telnet sshfs

Openocd installation

• Clon openocd repository and compile: shell=git clone git://git.code.sf.net/p/openocd/code openocd-codecd openocd-code./bootstrap./configure --enable-sysfsgpio --enable-bcm2835gpiomakesudo make installwzxhzdk:3You can store this file in OpenOCD scripts dir so it will auto find itshell= sudo mv sck.cfg /usr/local/share/openocd/scripts/

  and then run the OpenOCD server with:
  ```shell=
  sudo openocd -f sck.cfg
  

Then you can connect to OpenOCD, if you want to connect from an external computer, replace 127.0.0.1 with your Raspberry Pi IP address. shell=telnet 127.0.0.1 4444wzxhzdk:5Then you can connect to OpenOCD from your computer with:shell= telnet raspi_address 4444

### Uploading Arduino original bootloader

* Get the bootloader file [here](https://github.com/arduino/ArduinoCore-samd/tree/master/bootloaders/zero) and build it.

* Connect to OpenOCD server and run:
```shell=
reset halt
at91samd bootloader 0
at91samd chip-erase
program samd21_sam_ba.bin verify
at91samd bootloader 8192
reset run

If you don't see any error youre done!

Uploading SCK Firmware

• Install platformio, download and build SCK firmware
• Connect to OpenOCD server and run: `shell=reset haltflash write_image firmware.bin 8192verify_image firmware.bin 8192reset runreset run

GDB

General description

The purpose of a debugger such as GDB is to allow you to see what is going on “inside” another program while it executes—or what another program was doing at the moment it crashed.

GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act: * Start your program, specifying anything that might affect its behavior. * Make your program stop on specified conditions. * Examine what has happened, when your program has stopped. * Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.

Debugging session with Raspberry Pi as the OpenOCD server

Once your raspberry pi is setup with above instructions you can just do: ```shell= ssh pi@RaspberryAddress sudo openocd -f sck.cfg & cd /platformio_project/path arm-none-eabi-gdb ./pioenvs/zeroUSB/firmware.elf (gdb) target remote RaspberryAddress:3333 (gdb) monitor reset run

If you are using *platformio*, you need to modify the compiling option to avoid optimisation with -0g message to the compiler. In case you are not using *platformio*, activate verbose compiling output at Arduino IDE and find your compiled .elf directory.

[env:zeroUSB] platform = atmelsam board = zeroUSB framework = arduino build_flags = -Og

Now we are all set and ready to go. The debugger is waiting for instructions on the execution, which we detail below.

!!! info
    **Quick handy instructions inside GDB environment**
    1. *(gdb)* appears in every line and you don't have to type it each time
    2. In case you need to exit GDB, just type in `quit`, but remember always killing the process before, should you have a target running
    ```shell=
    (gdb) kill
    (gdb) quit
    ```
    3. `RET` repeats the previous command

### GDB commands

All commands in gdb during debugging are detailed in the GDB guide, chapter [GDB commands in detail (continue and stepping)](https://sourceware.org/gdb/onlinedocs/gdb/Continuing-and-Stepping.html)

An **extract** of some useful commands are detailed below:

#### Continuing and stepping

`continue [ignore-count]`

* Resumes program execution until next breakpoint. `[ignore-count]` argument allows to specify a further number of times to ingore a breakpoint.
```shell=
(gdb) continue
Continuing.

Breakpoint 1, tick () at src/HOLA.cpp:9
9   void tick() {

step count

• Continues running your program until control reaches a different source line, only availabe for source lines and functions compiled with debugging information. count is optional and states the number of steps to be performed before stopping, if no breakpoint arrives earlier.

next [count]

• Continue to the next source line without going into functions. It has the same functionality as step, but it stays in the same stack frame. count works as in step count. As well, it understands jumps calls as in the end of for loops and return to the beginning of the loop.

Info

set step-mode on/off sets the behaviour of (gdb) when stepping into a function with no debugging information. In the case of step-mode on, it inspects the first line of code of the function, whereas on step-mode off it skips the function completely.

finish

• Continue running until just after function in the selected stack frame returns.

until

• Has the same behaviour as step, but it ignores the jumps between lines due to loops (for, whiles, etc), continuing to the next source code with incremental line number.

Breakpoints

info breakpoints

• Retrieve information about breakpoints ```shell= (gdb) info breakpoints Num Type Disp Enb ress What 1 breakpoint keep y 0x00002140 in tick() at src/HOLA.cpp:9 breakpoint already hit 15 times

  `break`
  
  * Set a breakpoint in a specific function
  ```shell=
  (gdb) break loop
  

• Set a breakpoint in a specific line (344) ```shell= (gdb) break main.cpp:344

  !!! info
      Use the **tbreak** command instead of break if you want to stop the program once, and then remove the breakpoint. More **breakpoint condition** options can be found [**here**](https://sourceware.org/gdb/current/onlinedocs/gdb/Conditions.html#Conditions) you can find 
  
  `watchpoint`
  
  * Set a watchpoint [**watchpoint**](https://sourceware.org/gdb/current/onlinedocs/gdb/Set-Watchpoints.html#Set-Watchpoints) to only stop once a variable has a certain value.
  ```shell=
  (gdb) watch timer
  

Info

Type in info watchpoints to get information about watchpoints.

commands

• Set a list of actions related to the breakpoint: ```shell= break main.cpp:50 commands silent printf "count is %d\n",count cont end

  `delete`
  
  * Delete a breakpoint
  ```shell=
  (gdb) delete 1
  (gdb) info breakpoints
  No breakpoints or watchpoints.
  

Printing / setting variables and more

loop

• Read what is around a certain function ``shell=(gdb) l loop25 //while (!Serial) {26 //; // wait for serial port to connect. Needed for native USB port only27 //}28 }2930 void loop() {31 // put your main code here, to run repeatedly:32 Serial.println("HOLA");33 tick();34 Serial.println(millis());wzxhzdk:12set`

• Set variable to a certain value ```shell= (gdb) set timer = 0

  #### Target commands (load)
  `load filename offset`
  
  * `Load` it is meant to make filename (an executable) available for debugging on the remote system—by downloading it. `load` also records the filename symbol table in GDB, like the add-symbol-file command. The file is loaded at whatever address is specified in the executable, also into flash memory.
  
  ### Making changes in the code
  Anytime we make a change in the code, we don't need to reload the debugging session. We can easily do so by:
  
  1. Compile the code:
      a. Define Shell build in Sublime Text and configure a build sytem with:
  
      ```
      "shell_cmd": "cd .. && pio run"
      ```
  
      Then, everytime you hit Ctrl+B (Cmd+B) and you use your custom build system, it will automatically use this option.
  
      b. Or hit `pio run` in another terminal located in your project root directory
  
  2. In gdb, `load` file. This will reload the file defined at the beginning of your debugging session and upload it to the target
  ```shell=
  (gdb) load
  Loading section .text, size 0x2e50 lma 0x2000
  Loading section .ramfunc, size 0x60 lma 0x4e50
  Loading section .data, size 0x110 lma 0x4eb0
  Start address 0x2910, load size 12224
  Transfer rate: 3 KB/sec, 4074 bytes/write.
  

• Keep debugging

GDB Console

TUI

GDB has a console GUI option available with the command line option --tui In the upper frame you can see the code that's being executed.

GDB init file

From this example dashboard we can generate a custom .gdbinit file for the SCK which will be placed in the HOME directory... (ON GOING)

Info

Would be interesting to generate a custom option for production validation and one for internal debugging purposes

For references about where to locate the .gdbinit and more custom behaviour for gdb in general see here.

GDB from Sublime Text

Setup Platformio project with sublime Text Setup sublimeGDB

Reference

General GDB references and examples

Debugging with GDB - Book

Debugging example from GDB and OpenOCD

Arduino zero example

Additional notes from Platformio configuration

1. How to set other DEBUG FLAGS
2. About project configuration with Platformio init
3. Check here for building an *.ini file with custom build target for debugging and production.